Mixed reality interactions

ABSTRACT

Embodiments that relate to interacting with a physical object in a mixed reality environment via a head-mounted display are disclosed. In one embodiment a mixed reality interaction program identifies an object based on an image from captured by the display. An interaction context for the object is determined based on an aspect of the mixed reality environment. A profile for the physical object is queried to determine interaction modes for the object. A selected interaction mode is programmatically selected based on the interaction context. A user input directed at the object is received via the display and interpreted to correspond to a virtual action based on the selected interaction mode. The virtual action is executed with respect to a virtual object associated with the physical object to modify an appearance of the virtual object. The modified virtual object is then displayed via the display.

BACKGROUND

Searching for and retrieving relevant information using computingdevices is a common task. Typical computing devices that are used tofind and display information may include smartphones, tablets, notebookcomputers, and the like. Such computing devices, however, are typicallylimited to connecting a user to relevant information via dedicatedgraphical user interfaces and corresponding display elements, such asweb search portals, hyperlinks, etc. These well-known interfaces anddisplay elements generated by a corresponding computing device are tiedto the device, and generally embody little or no context linking them tothe user or the user's environment. Further, typical computing devicesare fairly limited in their capacity to understand an environment inwhich they are used and/or to link user activity within the environmentto information relevant to the user.

SUMMARY

Various embodiments are disclosed herein that relate to systems andmethods for interacting with a physical object in a mixed realityenvironment. For example, one disclosed embodiment provides a method forinteracting with a physical object in a mixed reality environment. Themethod includes providing a head-mounted display device operativelyconnected to a computing device, with the head-mounted display deviceincluding a display system for presenting the mixed reality environmentand a plurality of input sensors including a camera for capturing animage of the physical object. A physical object is identified based onthe captured image, and an interaction context is determined for theidentified physical object based on one or more aspects of the mixedreality environment.

The method includes querying a stored profile for the physical object todetermine a plurality of interaction modes for the physical object. Themethod includes programmatically selecting a selected interaction modefrom the plurality of interaction modes based on the interactioncontext. A user input directed at the physical object is received viaone of the input sensors of the head-mounted display device. The userinput is interpreted to correspond to a virtual action based on theselected interaction mode.

The method further includes executing the virtual action with respect toa virtual object that is associated with the physical object to therebymodify the virtual object's appearance. The method then displays thevirtual object via the head-mounted display device with the modifiedappearance.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a mixed reality interaction systemaccording to an embodiment of the present disclosure.

FIG. 2 shows an example head-mounted display device according to anembodiment of the present disclosure.

FIG. 3 is a schematic perspective view of a user wearing thehead-mounted display device of FIG. 2 and using the mixed realityinteraction system in an office according to an embodiment of thepresent disclosure.

FIGS. 4A, 4B and 4C are a flow chart of a method for interacting with aphysical object in a mixed reality environment according to anembodiment of the present disclosure.

FIG. 5 is a simplified schematic illustration of an embodiment of acomputing device.

DETAILED DESCRIPTION

FIG. 1 shows a schematic view of one embodiment of a mixed realityinteraction system 10. The mixed reality interaction system 10 includesa mixed reality interaction program 14 that may be stored in massstorage 18 of a computing device 22. The mixed reality interactionprogram 14 may be loaded into memory 28 and executed by a processor 30of the computing device 22 to perform one or more of the methods andprocesses described in more detail below.

The mixed reality interaction system 10 includes a mixed reality displayprogram 32 that may generate a virtual environment 34 for display via adisplay device, such as the head-mounted display (HMD) device 36, tocreate a mixed reality environment 38. As described in more detailbelow, the virtual environment 34 may include one or more virtualobjects, such as virtual object1 40, virtual object2 42, and geo-locatedtarget virtual object 52. Such virtual objects may include one or morevirtual images, such as three-dimensional holographic objects and othervirtual objects, as well as as two-dimensional virtual images, that aregenerated and displayed by HMD device 36.

The computing device 22 may take the form of a desktop computing device,a mobile computing device such as a smart phone, laptop, notebook ortablet computer, network computer, home entertainment computer,interactive television, gaming system, or other suitable type ofcomputing device. Additional details regarding the components andcomputing aspects of the computing device 22 are described in moredetail below with reference to FIG. 5.

The computing device 22 may be operatively connected with the HMD device36 using a wired connection, or may employ a wireless connection viaWiFi, Bluetooth, or any other suitable wireless communication protocol.For example, the computing device 22 may be communicatively coupled to anetwork 16. The network 16 may take the form of a local area network(LAN), wide area network (WAN), wired network, wireless network,personal area network, or a combination thereof, and may include theInternet.

As described in more detail below, the computing device 22 maycommunicate with one or more other computing devices, such as server 20,via network 16. Additionally, the example illustrated in FIG. 1 showsthe computing device 22 as a separate component from the HMD device 36.It will be appreciated that in other examples the computing device 22may be integrated into the HMD device 36.

With reference now also to FIG. 2, one example of an HMD device 200 inthe form of a pair of wearable glasses with a transparent display 44 isprovided. It will be appreciated that in other examples, the HMD device200 may take other suitable forms in which a transparent,semi-transparent or non-transparent display is supported in front of aviewer's eye or eyes. It will also be appreciated that the HMD device 36shown in FIG. 1 may take the form of the HMD device 200, as described inmore detail below, or any other suitable HMD device. Additionally, manyother types and configurations of display devices having various formfactors may also be used within the scope of the present disclosure.Such display devices may include hand-held smart phones, tabletcomputers, and other suitable display devices.

With reference to FIGS. 1 and 2, the HMD device 36 includes a displaysystem 48 and transparent display 44 that enables images such asholographic objects to be delivered to the eyes of a user 46. Thetransparent display 44 may be configured to visually augment anappearance of a physical environment 50 to a user 46 viewing thephysical environment through the transparent display. For example, theappearance of the physical environment 50 may be augmented by graphicalcontent (e.g., one or more pixels each having a respective color andbrightness) that is presented via the transparent display 44 to create amixed reality environment.

The transparent display 44 may also be configured to enable a user toview a physical, real-world object, such as physical object1 54 and/orphysical object2 58, in the physical environment 50 through one or morepartially transparent pixels that are displaying a virtual objectrepresentation. As shown in FIG. 2, in one example the transparentdisplay 44 may include image-producing elements located within lenses204 (such as, for example, a see-through Organic Light-Emitting Diode(OLED) display). As another example, the transparent display 44 mayinclude a light modulator on an edge of the lenses 204. In this examplethe lenses 204 may serve as a light guide for delivering light from thelight modulator to the eyes of a user. Such a light guide may enable auser to perceive a 3D holographic image located within the physicalenvironment 50 that the user is viewing, while also allowing the user toview physical objects in the physical environment, thus creating a mixedreality environment.

The HMD device 36 may also include various sensors and related systems.For example, the HMD device 36 may include an eye-tracking system 62that utilizes at least one inward facing sensor 216. The inward facingsensor 216 may be an image sensor that is configured to acquire imagedata in the form of eye-tracking data 66 from a user's eyes. Providedthe user has consented to the acquisition and use of this information,the eye-tracking system 62 may use this information to track a positionand/or movement of the user's eyes.

In one example, the eye-tracking system 62 includes a gaze detectionsubsystem configured to detect a direction of gaze of each eye of auser. The gaze detection subsystem may be configured to determine gazedirections of each of a user's eyes in any suitable manner. For example,the gaze detection subsystem may comprise one or more light sources,such as infrared light sources, configured to cause a glint of light toreflect from the cornea of each eye of a user. One or more image sensorsmay then be configured to capture an image of the user's eyes.

Images of the glints and of the pupils as determined from image datagathered from the image sensors may be used to determine an optical axisof each eye. Using this information, the eye-tracking system 62 may thendetermine a direction and/or at what physical object or virtual objectthe user is gazing. Such eye-tracking data 66 may then be provided tothe computing device 22. It will be understood that the gaze detectionsubsystem may have any suitable number and arrangement of light sourcesand image sensors.

The HMD device 36 may also include sensor systems that receive physicalenvironment data 60 from the physical environment 50. For example, theHMD device 36 may include an optical sensor system 68 that utilizes atleast one outward facing sensor 212, such as an optical sensor, tocapture image data 74. Outward facing sensor 212 may detect movementswithin its field of view, such as gesture-based inputs or othermovements performed by a user 46 or by a person or physical objectwithin the field of view. Outward facing sensor 212 may also capturetwo-dimensional image information and depth information from physicalenvironment 50 and physical objects within the environment. For example,outward facing sensor 212 may include a depth camera, a visible lightcamera, an infrared light camera, and/or a position tracking camera.

The HMD device 36 may include depth sensing via one or more depthcameras. In one example, each depth camera may include left and rightcameras of a stereoscopic vision system. Time-resolved images from oneor more of these depth cameras may be registered to each other and/or toimages from another optical sensor such as a visible spectrum camera,and may be combined to yield depth-resolved video.

In other examples a structured light depth camera may be configured toproject a structured infrared illumination, and to image theillumination reflected from a scene onto which the illumination isprojected. A depth map of the scene may be constructed based on spacingsbetween adjacent features in the various regions of an imaged scene. Instill other examples, a depth camera may take the form of atime-of-flight depth camera configured to project a pulsed infraredillumination onto a scene and detect the illumination reflected from thescene. It will be appreciated that any other suitable depth camera maybe used within the scope of the present disclosure.

Outward facing sensor 212 may capture images of the physical environment50 in which a user 46 is situated. In one example, the mixed realitydisplay program 32 may include a 3D modeling system that uses such inputto generate a virtual environment 34 that models the physicalenvironment 50 surrounding the user 46.

The HMD device 36 may also include a position sensor system 72 thatutilizes one or more motion sensors 220 to capture position data 76, andthereby enable motion detection, position tracking and/or orientationsensing of the HMD device. For example, the position sensor system 72may be utilized to determine a direction, velocity and acceleration of auser's head. The position sensor system 72 may also be utilized todetermine a head pose orientation of a user's head. In one example,position sensor system 72 may comprise an inertial measurement unitconfigured as a six-axis or six-degree of freedom position sensorsystem. This example position sensor system may, for example, includethree accelerometers and three gyroscopes to indicate or measure achange in location of the HMD device 36 within three-dimensional spacealong three orthogonal axes (e.g., x, y, z), and a change in anorientation of the HMD device about the three orthogonal axes (e.g.,roll, pitch, yaw).

Position sensor system 72 may also support other suitable positioningtechniques, such as GPS or other global navigation systems. Further,while specific examples of position sensor systems have been described,it will be appreciated that other suitable position sensor systems maybe used. In some examples, motion sensors 220 may also be employed asuser input devices, such that a user may interact with the HMD device 36via gestures of the neck and head, or even of the body. The HMD device36 may also include a microphone system 80 that includes one or moremicrophones 224 that capture audio data 82. In other examples, audio maybe presented to the user via one or more speakers 228 on the HMD device36.

The HMD device 36 may also include a processor 230 having a logicsubsystem and a storage subsystem, as discussed in more detail belowwith respect to FIG. 5, that are in communication with the varioussensors and systems of the HMD device. In one example, the storagesubsystem may include instructions that are executable by the logicsubsystem to receive signal inputs from the sensors and forward suchinputs to computing device 22 (in unprocessed or processed form), and topresent images to a user via the transparent display 44.

It will be appreciated that the HMD device 36 and related sensors andother components described above and illustrated in FIGS. 1 and 2 areprovided by way of example. These examples are not intended to belimiting in any manner, as any other suitable sensors, components,and/or combination of sensors and components may be utilized. Thereforeit is to be understood that the HMD device 36 may include additionaland/or alternative sensors, cameras, microphones, input devices, outputdevices, etc. without departing from the scope of this disclosure.Further, the physical configuration of the HMD device 36 and its varioussensors and subcomponents may take a variety of different forms withoutdeparting from the scope of this disclosure.

With reference now to FIG. 3, descriptions of example use cases andembodiments of the mixed reality interaction system 10 will now beprovided. FIG. 3 provides a schematic illustration of a user 304 locatedin a physical environment 50 that comprises an office 308, with the userexperiencing a mixed reality environment 38 via an HMD device 36 in theform of HMD device 200. As discussed in more detail below, in FIG. 3 themixed reality environment 38 may comprise the office 308 and one or morephysical objects, as well as a virtual environment including one or morevirtual objects.

In the example illustrated in FIG. 3, the user 304 sits at a desk 312that includes a framed photograph 316 of the user's spouse and akeyboard 320. The optical sensor system 68 of the HMD device 200 maycapture image data 74 from the office 308, including image datarepresenting the photograph 316 and other physical objects in theoffice, such as table 324, book 328 on the table, basketball 332,bookcase 334 and coat rack 338. Image data 74 of one or more of thesephysical objects may be provided by the HMD device 200 to the mixedreality interaction program 14.

Using this image data 74, the mixed reality interaction program 14 mayidentify one or more of these physical objects. For example, the mixedreality interaction program 14 may identify the photograph 316 and theface of the user's spouse in the photograph by comparing image data 74of the face to stored face image data and corresponding identitieslocated on server 20. In this example, the mixed reality interactionprogram 14 may include a face detection subsystem that detects faceimages in the image data 74. To detect a face image in the image data74, the face detection subsystem may use any suitable face detectiontechnologies and/or algorithms including local binary patterns (LBP),principal component analysis (PCA), independent component analysis(ICA), evolutionary pursuit (EP), Elastic Bunch Graph Matching (EBGM),or other suitable algorithm or combination of algorithms.

The face detection program may access object profile data 92 on server20 to match image data 74 including the face in photograph 316 with oneor more images and related user profile information corresponding touser's spouse. It will be appreciated that the face detection programmay use any suitable facial recognition techniques to match image data74 with stored images of the user's spouse.

The mixed reality interaction program 14 also determines an interactioncontext 84 for the framed photograph 316 based on one or more aspects ofthe mixed reality environment 38. With reference again to FIG. 1, suchaspects of the mixed reality environment 38 may include one or more datafeeds 86 originating from the mixed reality environment 38 or externallyto the mixed reality environment. Data feeds 86 may include temporaldata, such as time of day, day of week, month, season, etc., socialnetworking data, such as postings, status updates, etc. related to user304, location data, weather data, etc.

In some examples and with reference to FIG. 3, such aspects of the mixedreality environment 38 may also include information more directly linkedto the office 308. Such information may include the presence of one ormore other persons in the office 308, the identities of such persons, anambient light level, an ambient noise level, etc. In other examples,such aspects of the mixed reality environment 38 may also includeinformation related to the user 304, such as user biometric parametersincluding, for example, heart rate, pupillary response, hemoglobinsaturation, skin conductivity, respiration, perspiration, and brainwaveactivity.

In one example, the mixed reality interaction program determines that acurrent time of day is 10:30 am in the mixed reality environment 38. Itwill be appreciated that the time of day may be determined, for example,by an internal clock within computing device 22 or may be received froman external source via data feeds 86. Accordingly, in the presentexample the mixed reality interaction program 14 uses the current timeof day of 10:30 am to determine a Business-Time interaction context forthe photograph 316.

Having identified the face of the user's spouse in the photograph 316,the mixed reality interaction program 14 may then query a stored profileof the user's spouse in object profile data 92 to determine a pluralityof interaction modes 88 that are available for the photograph 316. Asdescribed in more detail below, each of the plurality of interactionmodes 88 may correspond to a different virtual action 90 that may betaken with respect to the photograph 316. In the present example, theavailable interaction modes 88 may include a Family Calendar interactionmode and a Family Reunion Planning interaction mode. In other examples,the mixed reality interaction program 14 may create a new profile forthe photograph 316 that associates one or more interaction modes withthe photograph. For example, the mixed reality interaction program 14may enable the user 304 to create a new profile for the photograph 316when, for example, a relationship status has changed.

Next, based on the determined interaction context, the mixed realityinteraction program 14 may programmatically select a selectedinteraction mode. In the present example, with the determinedinteraction context being Business-Time, the mixed reality interactionprogram 14 programmatically selects a Family Calendar interaction mode.In one example, the Business-Time interaction context may be defined asa time of day falling between 8:00 am and 6:00 pm. The mixed realityinteraction program 14 may be configured to pair the Business-Timeinteraction context with the Family Calendar interaction mode withrespect to the photograph 316.

Another interaction context, a Personal-Time interaction context, may bedefined as a time of day falling between 6:01 pm and 7:59 am. The mixedreality interaction program 14 may be configured to pair thePersonal-Time interaction context with a Family Reunion Planninginteraction mode with respect to the photograph 316.

In the present example, the mixed reality interaction program 14 mayreceive a user input from user 304 that is directed at the photograph316. For example, the mixed reality interaction program 14 may receiveeye-tracking data 66 from the HMD device 200 indicating that the user304 is gazing at the photograph 316, as indicated by gaze line 336. Inother examples, the user input may take one or more other formsincluding, for example, position data 76 and image data 74. The positiondata 76 may include head pose data indicating that the user 304 isfacing the photograph 316. The image data 74 may include image datashowing the user 304 pointing or gesturing at the photograph 316. Itwill be appreciated that other forms of user input may similarly beutilized.

The mixed reality interaction program 14 may then interpret the user 304gazing at the photograph 316 to correspond to a virtual action 90. Thevirtual action 90 may be based on the selected interaction mode, in thisexample the Family Calendar interaction mode. The virtual action 90 maycomprise presenting to the user 304 a virtual object in the form of theuser's family calendar stored in a calendar application. It will beappreciated that the user's family calendar is associated with thephotograph 316 of the user's spouse.

The mixed reality interaction program 14 may then execute the virtualaction 90 with respect to a virtual instantiation of the user's familycalendar to modify an appearance of the calendar. For example, tocomfortably display the user's family calendar to the user 304, themixed reality interaction program 14 may control the mixed realitydisplay program 32 to scale the virtual instantiation of the calendar toa predetermined size. The mixed reality interaction program 14 may thendisplay the virtual instantiation of the user's family calendar with themodified appearance, as indicated at 340, via the mixed reality displayprogram 32 and HMD device 200.

In other examples, the mixed reality interaction program 14 may enablethe user 304 to create and add new virtual actions 90 that areassociated with the photograph 316. For example, the user 304 may createa virtual action that includes highlighting family members' birthdays onthe family calendar. Thereafter, when the user 304 gazes at thephotograph 316, the mixed reality interaction program 14 will highlightfamily members' birthdays on the displayed family calendar.

In one example, the mixed reality interaction program 14 may geo-locatethe family calendar 340 in physical proximity to the photograph 316. Asshown in FIG. 3, in the present example the calendar 340 may bedisplayed just above the photograph 316 such that when the user 304gazes at the photograph, the calendar is presented in an easily viewablelocation just above the photograph. Being geo-located to the photograph316, the calendar 340 may remain “tethered” to the photograph and maytrack the location of the photograph in the office 308. Advantageously,if the photograph 316 is moved to another location in the office 308,the user 304 may still easily recall and view the calendar 340 by gazingat the photograph 316.

In another example, the user 304 may use his right hand 344 to point atthe basketball 332 near wall 348. The mixed reality interaction program14 may identify the basketball, determine an interaction context for thebasketball, query a stored profile for the basketball, andprogrammatically select a selected interaction mode based on theinteraction context in a manner similar to that described above for thephotograph 316. A camera in the HMD device 200 may capture image data 74showing the user's right hand 344 pointing at the basketball 332. Themixed reality interaction program 14 may interpret this user input tocorrespond to displaying a website of the user's favorite basketballteam.

The mixed reality interaction program 14 may then modify image data ofthe website to, for example, enhance the contrast of the displayedwebsite when displayed above the basketball and in front of the wall348. The mixed reality interaction program 14 may then display aholographic version of the modified website 352 above the basketball332. The website 352 may also be geo-located to the basketball 332 suchthat it follows the basketball to different locations in the office 308,as indicated by basketball 332′ and website 352′ located near the leftrear corner of the office.

In some examples, the mixed reality interaction program 14 may be unableto determine the identity of a physical object, or may determine anincomplete identity of the object. For example, the mixed realityinteraction program 14 may not recognize a visitor 342 who enters theoffice 308. The mixed reality interaction program 14 may also determinethat the user 304 currently has a meeting in the office 308 scheduled inthe user's calendar with a John Doe. Based on this meeting, the mixedreality interaction program 14 may then query the user 304, “Is thisJohn Doe?” via text displayed on the HMD device 200. The user 304 mayprovide a confirmation of the identity of the visitor 342 by, forexample, nodding, speaking “Yes”, or by any other suitable input.

In another example, where the user 304 does not confirm that the visitor342 is John Doe, the mixed reality interaction program 14 may presentanother query to the user requesting the identity of the visitor. Whenthe user 304 answers and provides an identity of the visitor, such as,“This is John Smith,” the program may then associate the identity JohnSmith with the visitor. It will also be appreciated that such identityconfirmation processes may be used with any other physical objects.

In another example, the mixed reality interaction program 14 maydetermine a change in the interaction context 84 and may correspondinglychange the interaction mode 86. For example, the user 304 may be workingin the office 308 late one night. At 7:01 pm the mixed realityinteraction program 14 determines that the interaction context changesfrom Business-Time to Personal-Time. Accordingly, the mixed realityinteraction program 14 may programmatically select the Family ReunionPlanning interaction mode with respect to the photograph 316.

The mixed reality interaction program 14 may then interpret the user 304gazing at the photograph 316 to correspond to a virtual action 90 thatis based on the Family Reunion Planning interaction mode. In thisexample, the virtual action 90 may comprise highlighting the dates 364of the user's family reunion gathering in the user's family calendar340. The mixed reality interaction program 14 may then execute thevirtual action with respect to the calendar 340 to modify an appearanceof the calendar. For example, the mixed reality interaction program 14may control the mixed reality display program 32 to highlight inflashing color the dates 364 of the family reunion gathering in thecalendar 340. The mixed reality interaction program 14 may then renderand display the calendar 340 with the highlighted dates 364 via themixed reality display program 32.

In another example, after determining a change in the interactioncontext 84 to the Family Reunion Planning interaction mode, the mixedreality interaction program 14 may execute a different virtual action 90with respect to a different virtual object that is associated with thephotograph 316. For example, the mixed reality interaction program 14may display a virtual family reunion To-Do List 360 via the mixedreality display program 32 and HMD device 200 to the user 304. As withthe family calendar 340, the mixed reality interaction program 14 maycontrol the mixed reality display program 32 to scale a virtualinstantiation of the family reunion To-Do List 360 to a predeterminedsize. The mixed reality interaction program 14 may then display theTo-Do List 360, via the mixed reality display program 32 with themodified appearance.

In another example, the user 304 may desire to manually switch betweenthe Family Calendar interaction mode and the Family Reunion Planninginteraction mode. The user may request that the current Family Calendarinteraction mode be modified by, for example, speaking “Switch to FamilyReunion Planning.” The mixed reality interaction program 14 interpretsthis user input as a request to modify the interaction mode, and changesthe interaction mode accordingly. The user may then point at thephotograph 316, which is captured as image data 74 via the HMD device200. The mixed reality interaction program 14 interprets this user inputas corresponding to a virtual action 90 based on the Family ReunionPlanning interaction mode, such as displaying the virtual family reunionTo-Do List 360. The mixed reality interaction program 14 may thenexecute the virtual action and display the family reunion To-Do List 360with a modified appearance as described above.

In another example, the mixed reality interaction program 14 may displaya target virtual object that may function to enable user interactionwith another virtual object in a manner similar to that described above.For example, the mixed reality interaction program 14 may display ageo-located target virtual object 52 in the form of a holographic wizard370 located on table 324 in the mixed reality environment 38. The mixedreality interaction program 14 may then determine a Business-Timeinteraction context for the wizard 370 based on one or more aspects ofthe mixed reality environment 38.

The mixed reality interaction program may query a stored profile of theholographic wizard 370 in the object profile data 92 to determine aplurality of interaction modes 88 that are available for the wizard. Inthe present example, the available interaction modes 88 may include aStock Market interaction mode and a Movie Listings interaction mode.Next, based on the determined interaction context, the mixed realityinteraction program 14 may programmatically select a selectedinteraction mode. In the present example, with the determinedinteraction context being Business-Time, the mixed reality interactionprogram 14 programmatically selects the Stock Market interaction mode.

In the present example, the mixed reality interaction program 14 mayreceive a user input from user 304 that is directed at the holographicwizard 370, such as position data 76 including head pose data indicatingthat the user's face is oriented toward the wizard 370. The mixedreality interaction program 14 may then interpret the user 304 facingthe wizard 370 to correspond to a virtual action 90 that is based on theselected interaction mode, in this example the Stock Market interactionmode, and is associated with the wizard 370. The virtual action 90 maycomprise presenting a stock market summary 374 via the mixed realitydisplay program 32 and HMD device 200 to the user 304.

The mixed reality interaction program 14 may then execute the virtualaction 90 with respect to the stock market summary to, for example,highlight quotes of stocks owned by the user 304, such as the XYZ stockquote 376. The mixed reality interaction program 14 may control themixed reality display program 32 to highlight the XYZ stock quote 376.The mixed reality interaction program 14 may then display the stockmarket summary 374 via the mixed reality display program 32.

In another example, the mixed reality interaction program 14 may alsouse a user input to create an association between a physical object anda selected interaction mode. With continued reference to FIG. 3, in oneexample the user 304 may desire to associate the basketball 332 with theAnytown Ants website 352. The user 304 may pick up the basketball 332and rotate the basketball in the user's hands.

Using image data 74, the mixed reality interaction program 14 mayrecognize the user's rotation of the basketball 332 as a trigger todisplay to the user 304 the available interaction modes 88 for thebasketball. Such interaction modes 88 may include, for example,displaying the Anytown Ants website, displaying a scoreboard of currentbasketball games, and displaying a calendar showing upcoming games inthe user's basketball league. The user 304 may then select the AnytownAnts website interaction mode to associate with the basketball 332 viauser input received by the HMD device 200. Accordingly, the mixedreality interaction program 14 may then be configured to associate thebasketball 332 with the Anytown Ants website interaction mode.Subsequently, the mixed reality interaction program 14 may execute avirtual action with respect to the Anytown Ants website that isassociated with the basketball 332 when user input directed at thebasketball 332 is received, as described above.

It will be appreciated that the present system may enhance a user'sability to associate particular information sources, portals, etc. withobjects in the mixed reality environment 38, whether physical objects orvirtual objects. For example, by linking the holographic wizard 370 witha displayed stock market summary, the user may more easily remember touse the wizard to see the stock market summary during business hours.The user may, for example, memorize the phrase, “I'm a Wizard of thestock market” to provide an easily-remembered cognitive link between theholographic wizard 370 in the user's office 308 and the stock marketsummary.

In another example, the mixed reality interaction program 14 maygenerate audio feedback that is perceived by the user 304 as originatingfrom a virtual object or physical object in the office 308. The mixedreality interaction program 14 may utilize a three-dimensional audiopositioning technique, such as a head-related transfer function, togenerate such audio feedback via speakers 228 of the HMD 200. In oneexample, the mixed reality interaction program 14 may generate audiocomprising a broadcast of an Anytown Ants basketball game that isperceived by the user 304 as originating from the virtual website 352 orthe basketball 332.

In another example, the mixed reality interaction program 14 may beconfigured to take a virtual action with respect to an object that isplaced on the table 324. With reference to FIG. 3, the mixed realityinteraction program 14 may recognize that book 328 is resting on thetable 324. Accordingly, based on the book 328 resting on the table 324,the mixed reality interaction program 14 may programmatically post astatus update to the user's social network that notes that the user 304is currently reading book 328.

In another example, the mixed reality interaction program 14 may beconfigured to take a virtual action when a state of an object changes.For example, the mixed reality interaction program 14 may identify thatthe user 304 has begun drinking his regular cup of morning coffee at6:30 am. The program may monitor the amount of coffee remaining in theuser's cup. If a predetermined amount of coffee is remaining in the cupand the user attempts to check email, the program may prevent ordissuade the user from checking email. For example, upon detecting anattempt to check email, the program may display a message to the userasking, “Wouldn't you like to finish your relaxing cup of coffee first?”

In another example, a document related to a work project managed by theuser 304 may be linked to the user, such as in the object profile data92. A fellow team member of the work project may also wear an HMD device200 that is communicatively coupled to a mixed reality interactionprogram 14. When the fellow team member sees the user 304, the teammember's mixed reality interaction program 14 recognizes the user 304and displays the document related to the work project via the teammembers' HMD device 200.

In another example, a team member may virtually deliver a document tothe user 304 via leaving a physical or virtual object on the user's desk312. For example, a team member of the user 304 may associate thedocument with a cardboard cutout of the letter “D.” The team member mayleave the cutout D on the user's desk 312 while the user 304 is away.Upon returning to the office 308, the user's HMD device 200 mayrecognize the cutout D. The HMD device 200 may indicate to the user 304that the cutout D is actively linked to virtual information by, forexample, animating the cutout D, such as by causing the cutout D to glowwhen viewed. The user 304 may request to view the linked virtualinformation, and the HMD device 200 may then access the document linkedto the cutout D, and may display the document for the user 304.

In another example, the user 304 may provide a user input that appliesto multiple virtual objects associated with multiple physical objects.For example and with reference to FIG. 3, the user 304 may desire to dimall of the virtual objects displayed in the office 308. The user 304 mayextend his right hand 344 outwardly and lower his hand toward the desk312. The mixed reality interaction system 10 may recognize this gesture,and may correspondingly dim all of the virtual objects displayed via HMD200.

In another example, when the user 304 gazes at the basketball 332, themixed reality interaction program 14 may interpret the user's gaze tocorrespond with displaying over the basketball a virtual volume controlknob. The volume control knob may be virtually rotated by the user 304to adjust the volume of the broadcast of the Anytown Ants basketballgame to which the user is listening. The current volume may also bedisplayed as a digital volume indicator 380. In another example, theuser 304 may pick up and physically rotate the basketball 332 tocorrespondingly adjust the volume of the broadcast via the virtualvolume control knob.

In another example, while the user 304 is away from the office 308, thevisitor 342 may pick up and rotate the basketball 332 to lower thevolume of the basketball game broadcast that is being delivered byspeakers in the room 308. When the user 304 returns to the office 308,the mixed reality interaction program 14 may identify the neworientation of the basketball 332 and correspondingly lower the volumeof the broadcast being delivered to the user via HMD 200. The programmay also correspondingly modify the volume indicator 380.

FIGS. 4A, 4B and 4C illustrate a flow chart of a method 400 forinteracting with a physical object in a mixed reality environmentaccording to an embodiment of the present disclosure. The followingdescription of method 400 is provided with reference to the software andhardware components of the mixed reality interaction system 10 describedabove and shown in FIGS. 1 and 2. It will be appreciated that method 400may also be performed in other contexts using other suitable hardwareand software components.

With reference to FIG. 4A, at 402 the method 400 includes providing anHMD device operatively connected to a computing device. The HMD deviceincludes a display system for presenting a mixed reality environment anda camera for capturing images of physical objects. At 406 the method 400includes identifying a physical object based on a captured image. At 410the method 400 may include presenting a query to the user to confirm anaccuracy of an identity of the object. At 412 the method 400 mayinclude, in response to the query, receiving confirmation of theaccuracy of the identity of the object.

At 414 the method 400 includes determining an interaction context forthe identified physical object based on one or more aspects of the mixedreality environment. At 416, the aspects of the mixed realityenvironment may comprise information received from one or more datafeeds. At 418 the method 400 includes querying a stored profile for thephysical object to determine a plurality of interaction modes for theobject. At 420 the method 400 includes programmatically selecting aselected interaction mode from the plurality of interaction modes basedon the interaction context. At 422 the method 400 includes receiving auser input directed at the physical object via one of the input sensorsof the HMD device. At 424 the method 400 includes interpreting the userinput to correspond to a virtual action based on the selectedinteraction mode.

At 426 the method 400 includes executing the virtual action with respectto a virtual object associated with the physical object to therebymodify an appearance of the virtual object. And at 428 the method 400includes displaying the virtual object via the HMD device with themodified appearance. With reference now to FIG. 4B, at 430 and in someexamples the method 400 includes geo-locating the virtual object inphysical proximity to the physical object. At 432 the method 400 mayinclude using a second user input to associate the physical object witha selected interaction mode.

In another example, at 434 the method 400 may include, where theselected interaction mode is a first selected interaction mode and thevirtual action is a first virtual action, determining a change in theinteraction context. At 436 the method 400 may include, based on thechange, programmatically selecting a second selected interaction modefrom a plurality of interaction mode. At 438 the method 400 may includeinterpreting user input to correspond to a second virtual action basedon the second selected interaction mode. At 440 the method 400 mayinclude executing the second virtual action with respect to the virtualobject associated with the physical object to modify the appearance ofthe virtual object. And at 442 the method 400 may include displaying thevirtual object via the HMD device with the modified appearance.

In another example, at 444 the method 400 may include executing a secondvirtual action with respect to a second virtual object that isassociated with the physical object to modify the appearance of thesecond virtual object. At 446 the method 400 may include displaying thesecond virtual object via the HMD device with the modified appearance.

In another example, at 448 the method 400 may include receiving a seconduser input via an input sensor of the HMD device. At 450 the method 400may include, where the selected interaction mode is a first selectedinteraction mode, interpreting the second user input to correspond to arequest to modify the first selected interaction mode. With referencenow to FIG. 4C, at 452 the method 400 includes, based on the second userinput, changing from the first selected interaction mode to a secondselected interaction mode. At 454 the method 400 may include receiving athird user input via the input sensor of the HMD device. At 456 themethod 400 may include interpreting the third user input to correspondto a second virtual action based on the second selected interactionmode.

At 458 the method 400 may include executing the second virtual actionwith respect to the virtual object associated with the physical objectto modify the appearance of the virtual object. At 460 the method mayinclude displaying the virtual object via the HMD device with themodified appearance.

In another example, at 462 the method 400 may include displaying ageo-located target virtual object in the mixed reality environment. At464 the method 400 may include determining an interaction context forthe geo-located target virtual object based on one or more aspects ofthe mixed reality environment. At 466 the method may include querying astored profile for the geo-located target virtual object to determine aplurality of interaction modes for the geo-located target virtualobject. At 468 the method 400 may include programmatically selecting aselected interaction mode for the geo-located target virtual objectbased on the interaction context. At 470 the method 400 may includereceiving a second user input directed at the geo-located target virtualobject via an input sensor of the HMD device.

At 472 the method may include interpreting the second user input tocorrespond to a second virtual action based on the selected interactionmode. At 474 the method 400 may include executing the second virtualaction with respect to the geo-located target virtual object to modifythe appearance of the geo-located target virtual object. And at 476 themethod 400 may include displaying the geo-located target virtual objectvia the HMD device with the modified appearance.

It will be appreciated that method 400 is provided by way of example andis not meant to be limiting. Therefore, it is to be understood thatmethod 400 may include additional and/or alternative steps than thoseillustrated in FIGS. 4A, 4B AND 4C. Further, it is to be understood thatmethod 400 may be performed in any suitable order. Further still, it isto be understood that one or more steps may be omitted from method 400without departing from the scope of this disclosure.

FIG. 5 schematically shows a nonlimiting embodiment of a computingsystem 500 that may perform one or more of the above described methodsand processes. Computing device 22 may take the form of computing system500. Computing system 500 is shown in simplified form. It is to beunderstood that virtually any computer architecture may be used withoutdeparting from the scope of this disclosure. In different embodiments,computing system 500 may take the form of a mainframe computer, servercomputer, desktop computer, laptop computer, tablet computer, homeentertainment computer, network computing device, mobile computingdevice, mobile communication device, gaming device, etc. As noted above,in some examples the computing system 500 may be integrated into an HMDdevice.

As shown in FIG. 5, computing system 500 includes a logic subsystem 504and a storage subsystem 508. Computing system 500 may optionally includea display subsystem 512, a communication subsystem 516, a sensorsubsystem 520, an input subsystem 522 and/or other subsystems andcomponents not shown in FIG. 5. Computing system 500 may also includecomputer readable media, with the computer readable media includingcomputer readable storage media and computer readable communicationmedia. Computing system 500 may also optionally include other user inputdevices such as keyboards, mice, game controllers, and/or touch screens,for example. Further, in some embodiments the methods and processesdescribed herein may be implemented as a computer application, computerservice, computer API, computer library, and/or other computer programproduct in a computing system that includes one or more computers.

Logic subsystem 504 may include one or more physical devices configuredto execute one or more instructions. For example, the logic subsystem504 may be configured to execute one or more instructions that are partof one or more applications, services, programs, routines, libraries,objects, components, data structures, or other logical constructs. Suchinstructions may be implemented to perform a task, implement a datatype, transform the state of one or more devices, or otherwise arrive ata desired result.

The logic subsystem 504 may include one or more processors that areconfigured to execute software instructions. Additionally oralternatively, the logic subsystem may include one or more hardware orfirmware logic machines configured to execute hardware or firmwareinstructions. Processors of the logic subsystem may be single core ormulticore, and the programs executed thereon may be configured forparallel or distributed processing. The logic subsystem may optionallyinclude individual components that are distributed throughout two ormore devices, which may be remotely located and/or configured forcoordinated processing. One or more aspects of the logic subsystem maybe virtualized and executed by remotely accessible networked computingdevices configured in a cloud computing configuration.

Storage subsystem 508 may include one or more physical, persistentdevices configured to hold data and/or instructions executable by thelogic subsystem 504 to implement the herein described methods andprocesses. When such methods and processes are implemented, the state ofstorage subsystem 508 may be transformed (e.g., to hold different data).

Storage subsystem 508 may include removable media and/or built-indevices. Storage subsystem 508 may include optical memory devices (e.g.,CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory devices(e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices (e.g.,hard disk drive, floppy disk drive, tape drive, MRAM, etc.), amongothers. Storage subsystem 508 may include devices with one or more ofthe following characteristics: volatile, nonvolatile, dynamic, static,read/write, read-only, random access, sequential access, locationaddressable, file addressable, and content addressable.

In some embodiments, aspects of logic subsystem 504 and storagesubsystem 508 may be integrated into one or more common devices throughwhich the functionally described herein may be enacted, at least inpart. Such hardware-logic components may include field-programmable gatearrays (FPGAs), program- and application-specific integrated circuits(PASIC/ASICs), program- and application-specific standard products(PSSP/ASSPs), system-on-a-chip (SOC) systems, and complex programmablelogic devices (CPLDs), for example.

FIG. 5 also shows an aspect of the storage subsystem 508 in the form ofremovable computer readable storage media 524, which may be used tostore data and/or instructions executable to implement the methods andprocesses described herein. Removable computer-readable storage media524 may take the form of CDs, DVDs, HD-DVDs, Blu-Ray Discs, EEPROMs,and/or floppy disks, among others.

It is to be appreciated that storage subsystem 508 includes one or morephysical, persistent devices. In contrast, in some embodiments aspectsof the instructions described herein may be propagated in a transitoryfashion by a pure signal (e.g., an electromagnetic signal, an opticalsignal, etc.) that is not held by a physical device for at least afinite duration. Furthermore, data and/or other forms of informationpertaining to the present disclosure may be propagated by a pure signalvia computer-readable communication media.

When included, display subsystem 512 may be used to present a visualrepresentation of data held by storage subsystem 508. As the abovedescribed methods and processes change the data held by the storagesubsystem 508, and thus transform the state of the storage subsystem,the state of the display subsystem 512 may likewise be transformed tovisually represent changes in the underlying data. The display subsystem512 may include one or more display devices utilizing virtually any typeof technology. Such display devices may be combined with logic subsystem504 and/or storage subsystem 508 in a shared enclosure, or such displaydevices may be peripheral display devices. The display subsystem 512 mayinclude, for example, the display system 48 and transparent display 44of the HMD device 36.

When included, communication subsystem 516 may be configured tocommunicatively couple computing system 500 with one or more networksand/or one or more other computing devices. Communication subsystem 516may include wired and/or wireless communication devices compatible withone or more different communication protocols. As nonlimiting examples,the communication subsystem 516 may be configured for communication viaa wireless telephone network, a wireless local area network, a wiredlocal area network, a wireless wide area network, a wired wide areanetwork, etc. In some embodiments, the communication subsystem may allowcomputing system 500 to send and/or receive messages to and/or fromother devices via a network such as the Internet.

Sensor subsystem 520 may include one or more sensors configured to sensedifferent physical phenomenon (e.g., visible light, infrared light,sound, acceleration, orientation, position, etc.) as described above.Sensor subsystem 520 may be configured to provide sensor data to logicsubsystem 504, for example. As described above, such data may includeeye-tracking information, image information, audio information, ambientlighting information, depth information, position information, motioninformation, user location information, and/or any other suitable sensordata that may be used to perform the methods and processes describedabove.

When included, input subsystem 522 may comprise or interface with one ormore sensors or user-input devices such as a game controller, gestureinput detection device, voice recognizer, inertial measurement unit,keyboard, mouse, or touch screen. In some embodiments, the inputsubsystem 522 may comprise or interface with selected natural user input(NUI) componentry. Such componentry may be integrated or peripheral, andthe transduction and/or processing of input actions may be handled on-or off-board. Example NUI componentry may include a microphone forspeech and/or voice recognition; an infrared, color, stereoscopic,and/or depth camera for machine vision and/or gesture recognition; ahead tracker, eye tracker, accelerometer, and/or gyroscope for motiondetection and/or intent recognition; as well as electric-field sensingcomponentry for assessing brain activity.

The term “program” may be used to describe an aspect of the mixedreality interaction system 10 that is implemented to perform one or moreparticular functions. In some cases, such a program may be instantiatedvia logic subsystem 504 executing instructions held by storage subsystem508. It is to be understood that different programs may be instantiatedfrom the same application, service, code block, object, library,routine, API, function, etc. Likewise, the same program may beinstantiated by different applications, services, code blocks, objects,routines, APIs, functions, etc. The term “program” is meant to encompassindividual or groups of executable files, data files, libraries,drivers, scripts, database records, etc.

It is to be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated may beperformed in the sequence illustrated, in other sequences, in parallel,or in some cases omitted. Likewise, the order of the above-describedprocesses may be changed.

The subject matter of the present disclosure includes all novel andnonobvious combinations and subcombinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A mixed reality interaction system for interacting with a physicalobject in a mixed reality environment, the mixed reality interactionsystem comprising: a head-mounted display device operatively connectedto a computing device, the head-mounted display device including adisplay system for presenting the mixed reality environment and aplurality of input sensors including a camera for capturing an image ofthe physical object; and a mixed reality interaction program executed bya processor of the computing device, the mixed reality interactionprogram configured to: identify the physical object based on thecaptured image; determine an interaction context for the identifiedphysical object based on one or more aspects of the mixed realityenvironment; query a stored profile for the physical object to determinea plurality of interaction modes for the physical object;programmatically select a selected interaction mode from the pluralityof interaction modes based on the interaction context; receive a userinput directed at the physical object via one of the input sensors ofthe head-mounted display device; interpret the user input to correspondto a virtual action based on the selected interaction mode; execute thevirtual action with respect to a virtual object associated with thephysical object to thereby modify an appearance of the virtual object;and display the virtual object via the head-mounted display device withthe modified appearance.
 2. The mixed reality interaction system ofclaim 1, wherein the mixed reality interaction program is furtherconfigured to: present a first query to confirm an accuracy of anidentity of the physical object; and in response to the query, either(1) receive a confirmation of the accuracy of the identity of thephysical object or (2) where the confirmation of the accuracy of theidentity is not received, present a second query requesting the identityof the physical object.
 3. The mixed reality interaction system of claim1, wherein the mixed reality interaction program is further configuredto geo-locate the virtual object in physical proximity to the physicalobject.
 4. The mixed reality interaction system of claim 1, wherein theselected interaction mode is a first selected interaction mode and thevirtual action is a first virtual action, and the mixed realityinteraction program is further configured to: determine a change in theinteraction context; based on the change in the interaction context,programmatically select a second selected interaction mode from theplurality of interaction modes; interpret the user input to correspondto a second virtual action based on the second selected interactionmode; execute the second virtual action with respect to the virtualobject associated with the physical object to thereby modify anappearance of the virtual object; and display the virtual object via thehead-mounted display device with the modified appearance.
 5. The mixedreality interaction system of claim 4, wherein the virtual object is afirst virtual object, and the mixed reality interaction program isfurther configured to: execute the second virtual action with respect toa second virtual object associated with the physical object to therebymodify an appearance of the second virtual object; and display thesecond virtual object via the head-mounted display device with themodified appearance.
 6. The mixed reality interaction system of claim 5,wherein the mixed reality interaction program is further configured to:interpret the user input to correspond to a third virtual actionapplicable to the first virtual object and the second virtual object;execute the third virtual action with respect to the first virtualobject and the second virtual object to thereby modify the appearance ofthe first virtual object and the second virtual object; and display thefirst virtual object and the second virtual object via the head-mounteddisplay device with the modified appearances.
 7. The mixed realityinteraction system of claim 1, wherein the selected interaction mode isa first selected interaction mode, the user input is a first user input,and the virtual action is a first virtual action, the mixed realityinteraction program is further configured to: receive a second userinput via one of the plurality of input sensors of the head-mounteddisplay device; interpret the second user input to correspond to arequest to modify the first selected interaction mode; based on thesecond user input, change from the first selected interaction mode to asecond selected interaction mode from the plurality of interactionmodes; receive a third user input via one of the plurality of inputsensors of the head-mounted display device; interpret the third userinput to correspond to a second virtual action based on the secondselected interaction mode; execute the second virtual action withrespect to the virtual object associated with the physical object tothereby modify an appearance of the virtual object; and display thevirtual object via the head-mounted display device with the modifiedappearance.
 8. The mixed reality interaction system of claim 1, whereinthe user input is a first user input and the virtual action is a firstvirtual action, wherein the mixed reality interaction program is furtherconfigured to: display a geo-located target virtual object in the mixedreality environment; determine an interaction context for thegeo-located target virtual object based on the one or more aspects ofthe mixed reality environment; query a stored profile for thegeo-located target virtual object to determine a plurality ofinteraction modes for the geo-located target virtual object;programmatically select a selected interaction mode from the pluralityof interaction modes for the geo-located target virtual object based onthe interaction context; receive a second user input directed at thegeo-located target virtual object via one of the input sensors of thehead-mounted display device; interpret the second user input tocorrespond to a second virtual action based on the selected interactionmode for the geo-located target virtual object; execute the secondvirtual action with respect to the geo-located target virtual object tothereby modify an appearance of the geo-located target virtual object;and display the geo-located target virtual object with the modifiedappearance.
 9. The mixed reality interaction system of claim 1, whereinthe mixed reality interaction program is further configured to create anew profile for the physical object that associates one or moreinteraction modes with the physical object.
 10. The mixed realityinteraction system of claim 1, wherein the mixed reality interactionprogram is further configured to: identify an updated physical state ofthe physical object; and based on the updated physical state, furthermodify the appearance of the virtual object associated with the physicalobject.
 11. A method for interacting with a physical object in a mixedreality environment, comprising: providing a head-mounted display deviceoperatively connected to a computing device, the head-mounted displaydevice including a display system for presenting the mixed realityenvironment and a plurality of input sensors including a camera forcapturing an image of the physical object; identifying the physicalobject based on the captured image; determining an interaction contextfor the identified physical object based on one or more aspects of themixed reality environment; querying a stored profile for the physicalobject to determine a plurality of interaction modes for the physicalobject; programmatically selecting a selected interaction mode from theplurality of interaction modes based on the interaction context;receiving a user input directed at the physical object via one of theinput sensors of the head-mounted display device; interpreting the userinput to correspond to a virtual action based on the selectedinteraction mode; executing the virtual action with respect to a virtualobject associated with the physical object to thereby modify anappearance of the virtual object; and displaying the virtual object viathe head-mounted display device with the modified appearance.
 12. Themethod of claim 11, further comprising: presenting a query to confirm anaccuracy of an identity of the physical object; and in response to thequery, receiving a confirmation of the accuracy of the identity of thephysical object.
 13. The method of claim 11, further comprisinggeo-locating the virtual object in physical proximity to the physicalobject.
 14. The method of claim 11, wherein the selected interactionmode is a first selected interaction mode and the virtual action is afirst virtual action, and the method further comprises: determining achange in the interaction context; based on the change in theinteraction context, programmatically selecting a second selectedinteraction mode from the plurality of interaction modes; interpretingthe user input to correspond to a second virtual action based on thesecond selected interaction mode; executing the second virtual actionwith respect to the virtual object associated with the physical objectto thereby modify an appearance of the virtual object; and displayingthe virtual object via the head-mounted display device with the modifiedappearance.
 15. The method of claim 11, wherein the virtual object is afirst virtual object, and the method further comprises: executing thesecond virtual action with respect to a second virtual object associatedwith the physical object to thereby modify an appearance of the secondvirtual object; and displaying the second virtual object via thehead-mounted display device with the modified appearance.
 16. The methodof claim 11, wherein the selected interaction mode is a first selectedinteraction mode, the user input is a first user input, and the virtualaction is a first virtual action, and the method further comprises:receiving a second user input via one of the plurality of input sensorsof the head-mounted display device; interpreting the second user inputto correspond to a request to modify the first selected interactionmode; based on the second user input, changing from the first selectedinteraction mode to a second selected interaction mode from theplurality of interaction modes; receiving a third user input via one ofthe plurality of input sensors of the head-mounted display device;interpreting the third user input to correspond to the second virtualaction based on the second selected interaction mode; executing thesecond virtual action with respect to the virtual object associated withthe physical object to thereby modify an appearance of the virtualobject; and displaying the virtual object via the head-mounted displaydevice with the modified appearance.
 17. The method of claim 11, whereinthe user input is a first user input and the virtual action is a firstvirtual action, and the method further comprises: displaying ageo-located target virtual object in the mixed reality environment;determining an interaction context for the geo-located target virtualobject based on the one or more aspects of the mixed realityenvironment; querying a stored profile for the geo-located targetvirtual object to determine a plurality of interaction modes for thegeo-located target virtual object; programmatically selecting a selectedinteraction mode from the plurality of interaction modes for thegeo-located target virtual object based on the interaction context;receiving a second user input directed at the geo-located target virtualobject via one of the input sensors of the head-mounted display device;interpreting the second user input to correspond to a second virtualaction based on the selected interaction mode for the geo-located targetvirtual object; executing the second virtual action with respect to thegeo-located target virtual object to thereby modify an appearance of thegeo-located target virtual object; and displaying the geo-located targetvirtual object with the modified appearance.
 18. The method of claim 11,wherein the user input is a first user input, and the method furthercomprises using a second user input to associate the physical objectwith the selected interaction mode.
 19. A method for interacting with aphysical object in a mixed reality environment, comprising: providing ahead-mounted display device operatively connected to a computing device,the head-mounted display device including a display system forpresenting the mixed reality environment and a plurality of inputsensors including a camera for capturing an image of the physicalobject; identifying the physical object based on the captured image;determining an interaction context for the identified physical objectbased on one or more aspects of the mixed reality environment; queryinga stored profile for the physical object to determine a plurality ofinteraction modes for the physical object; programmatically selecting afirst selected interaction mode from the plurality of interaction modesbased on the interaction context; receiving a first user input directedat the physical object via one of the input sensors of the head-mounteddisplay device; interpreting the first user input to correspond to afirst virtual action based on the first selected interaction mode;executing the first virtual action with respect to a virtual objectassociated with the physical object to thereby modify an appearance ofthe virtual object to a first modified appearance; and displaying thevirtual object via the head-mounted display device with the firstmodified appearance; determining a change in the interaction context;based on the change in the interaction context, programmaticallyselecting a second selected interaction mode from the plurality ofinteraction modes; receiving a second user input directed at thephysical object via one of the input sensors of the head-mounted displaydevice; interpreting the second user input to correspond to a secondvirtual action based on the second selected interaction mode; executingthe second virtual action with respect to the virtual object associatedwith the physical object to thereby modify an appearance of the virtualobject to a second modified appearance; and displaying the virtualobject via the head-mounted display device with the second modifiedappearance.
 20. The method of claim 19, wherein the first user input andthe second user input are each selected from the group consisting ofeye-tracking input, voice input, gesture input, and head movement input.